Mobile visual-inspection system

ABSTRACT

A ground-based visual-inspection system includes a ground-based visual-inspection apparatus and a control system. The ground-based visual-inspection apparatus includes a mobile base, an actuatable arm coupled to the mobile base, and an effector coupled to the actuatable arm. The actuatable arm is locatable in a three dimensional space. The end effector includes a camera configured to capture images of a structure, such as an aircraft. The control system is configured to determine location information of the camera relative to a reference location and associate the location information with the images.

FIELD

This disclosure relates generally to visual-inspection systems, and moreparticularly to ground-based mobile visual-inspection systems forinspecting large structures, such as aircraft.

BACKGROUND

Aircraft and other mobile vehicles sometimes require inspection forwear. One type of inspection is visual-inspection. For large vehicles,visual-inspection can be difficult to accomplish. For many hard to reachareas, drones or other unmanned aerial vehicles are used to fly camerasclose to the aircraft. However, drones have vibrating parts and may beadversely affected by weather conditions, such as wind or precipitation.To prevent the drones from impacting the aircraft, the drones are flowna safe distance from the aircraft, which results in less than idealphotographs and video of the aircraft.

SUMMARY

The subject matter of the present application has been developed inresponse to the present state of the art, and in particular, in responseto the problems and disadvantages associated with conventional fixturesthat have not yet been fully solved by currently available techniques.Accordingly, the subject matter of the present application has beendeveloped to provide embodiments of a system, an apparatus, and a methodthat overcome at least some of the above-discussed shortcomings of priorart techniques. For example, according to one implementation,ground-based mobile visual-inspection system is disclosed, whichfacilitates the capture of steady and precisely-located photos andvideos.

Disclosed herein is a ground-based visual-inspection system including aground-based visual-inspection apparatus and a control system. Theground-based visual-inspection apparatus includes a mobile base, anactuatable arm coupled to the mobile base, and an effector coupled tothe actuatable arm. The actuatable arm is locatable in a threedimensional space. The end effector includes a camera configured tocapture images of a structure. The control system is configured todetermine location information of the camera relative to a referencelocation and associate the location information with the images. Thepreceding subject matter of this paragraph characterizes example 1 ofthe present disclosure.

The control system is configured to determine the location informationby determining a location of the camera relative to the referencelocation by acquiring a base location by computing a transformationmatrix of the mobile base relative to the reference location, acquiringan arm location by computing a transformation matrix of the actuatablearm relative to the reference location, and performing matrixmultiplication of the transformation matrix of the mobile base and thetransformation matrix of the actuatable arm to compute a transformationmatrix of the end effector relative to the reference location. Thepreceding subject matter of this paragraph characterizes example 2 ofthe present disclosure, wherein example 2 also includes the subjectmatter according to example 1, above.

The control system further includes an alignment system including aplurality of laser rangefinders coupled to the end effector. Thepreceding subject matter of this paragraph characterizes example 3 ofthe present disclosure, wherein example 3 also includes the subjectmatter according to any one of examples 1-2, above.

The alignment system is configured to determine an orientation of thecamera relative to the structure based on data of the plurality of laserrangefinders. The preceding subject matter of this paragraphcharacterizes example 4 of the present disclosure, wherein example 4also includes the subject matter according to example 3, above.

The control system is further configured to embed orientationinformation of the camera in the images. The preceding subject matter ofthis paragraph characterizes example 5 of the present disclosure,wherein example 5 also includes the subject matter according to any oneof examples 1-4, above.

The alignment system is configured to determine a distance of the camerarelative to the structure based on data of the plurality of laserrangefinders. The preceding subject matter of this paragraphcharacterizes example 6 of the present disclosure, wherein example 6also includes the subject matter according to any one of examples 3-5,above.

The control system is further configured to embed distance informationin the images. The preceding subject matter of this paragraphcharacterizes example 7 of the present disclosure, wherein example 7also includes the subject matter according to any one of examples 1-6,above.

The control system is configured to embed the location information intothe image. The preceding subject matter of this paragraph characterizesexample 8 of the present disclosure, wherein example 8 also includes thesubject matter according to any one of examples 1-7, above.

The location information is based on data from sensors on theground-based visual-inspection apparatus. The preceding subject matterof this paragraph characterizes example 9 of the present disclosure,wherein example 9 also includes the subject matter according to any oneof examples 1-8, above.

The location information is based on a reference position on thestructure. The preceding subject matter of this paragraph characterizesexample 10 of the present disclosure, wherein example 10 also includesthe subject matter according to any one of examples 1-9, above.

The mobile base is a cart maneuverable by manual power. The precedingsubject matter of this paragraph characterizes example 11 of the presentdisclosure, wherein example 11 also includes the subject matteraccording to any one of examples 1-10, above.

The system further includes an alignment system including a plurality oflaser rangefinders coupled to the end effector, wherein the alignmentsystem is configured to determine an orientation and a distance of thecamera relative to the structure based on data of the plurality of laserrangefinders, wherein the control system is further configured to embedorientation information and distance information with the images. Thecontrol system is configured to embed the location information into theimage. The preceding subject matter of this paragraph characterizesexample 12 of the present disclosure, wherein example 12 also includesthe subject matter according to any one of examples 1-11, above.

The location information includes a translational displacement from areference position and rotational displacement from a referenceorientation of the mobile base relative to the reference location. Thepreceding subject matter of this paragraph characterizes example 13 ofthe present disclosure, wherein example 13 also includes the subjectmatter according to any one of examples 1-12, above.

The actuatable arm includes an indexing rail. The preceding subjectmatter of this paragraph characterizes example 14 of the presentdisclosure, wherein example 14 also includes the subject matteraccording to any one of examples 1-13, above.

Also disclosed herein is a ground-based visual-inspection apparatus. Theground-based visual-inspection apparatus includes a mobile base, anactuatable arm coupled to the mobile base, an effector coupled to theactuatable arm, and a control system. The actuatable arm is locatable ina three dimensional space. The end effector includes a camera configuredto capture images of a structure. The control system is configured todetermine location information of the camera relative to a referencelocation. The preceding subject matter of this paragraph characterizesexample 15 of the present disclosure.

The ground-based visual-inspection apparatus further includes aplurality of laser rangefinders coupled to the end effector. Thepreceding subject matter of this paragraph characterizes example 16 ofthe present disclosure, wherein example 16 also includes the subjectmatter according to example 15, above.

Also disclosed herein is a method. The method includes aligning aground-based visual-inspection apparatus to a structure, theground-based visual-inspection apparatus comprising a mobile base, anactuatable arm coupled to the mobile base, and an end effector coupledto the actuatable arm, wherein the end effector includes a camera. Themethod also includes capturing an image of the structure with the cameraand determining location information of the camera relative to areference location by acquiring a base location, arm location, and endeffector location relative to a reference location. The method alsoincludes associating the location information with the image. Thepreceding subject matter of this paragraph characterizes example 17 ofthe present disclosure.

The method further includes receiving data from sensors andautomatically adjusting the orientation or distance of the camerarelative to the structure based on the received data. The precedingsubject matter of this paragraph characterizes example 18 of the presentdisclosure, wherein example 18 also includes the subject matteraccording to example 17, above.

The method further includes acquiring the base location by computing atransformation matrix of the mobile base relative to the referencelocation, acquiring the arm location by computing a transformationmatrix of the actuatable arm relative to the reference location, andperforming matrix multiplication of the transformation matrix of themobile base and the transformation matrix of the actuatable arm tocompute a transformation matrix of the end effector relative to thereference location. The preceding subject matter of this paragraphcharacterizes example 19 of the present disclosure, wherein example 19also includes the subject matter according to any one of examples 17-18,above.

The method further includes determining distance information andorientation information of the camera relative to the structure andassociating the distance information and the orientation informationwith the image. The preceding subject matter of this paragraphcharacterizes example 20 of the present disclosure, wherein example 20also includes the subject matter according to any one of examples 17-19,above.

The described features, structures, advantages, and/or characteristicsof the subject matter of the present disclosure may be combined in anysuitable manner in one or more embodiments and/or implementations. Inthe following description, numerous specific details are provided toimpart a thorough understanding of embodiments of the subject matter ofthe present disclosure. One skilled in the relevant art will recognizethat the subject matter of the present disclosure may be practicedwithout one or more of the specific features, details, components,materials, and/or methods of a particular embodiment or implementation.In other instances, additional features and advantages may be recognizedin certain embodiments and/or implementations that may not be present inall embodiments or implementations. Further, in some instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring aspects of the subject matter ofthe present disclosure. The features and advantages of the subjectmatter of the present disclosure will become more fully apparent fromthe following description and appended claims, or may be learned by thepractice of the subject matter as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the subject matter may be more readilyunderstood, a more particular description of the subject matter brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict only typical embodiments of the subject matter and arenot therefore to be considered to be limiting of its scope, the subjectmatter will be described and explained with additional specificity anddetail through the use of the drawings, in which:

FIG. 1 is a side view of a ground-based visual-inspection apparatus,according to one or more embodiments of the present disclosure;

FIG. 2 is a side view of a ground-based visual-inspection apparatus inan inspecting position relative to an aircraft, according to one or moreembodiments of the present disclosure;

FIG. 3A is a perspective view of an end effector with a camera in amisaligned orientation, according to one or more embodiments of thepresent disclosure;

FIG. 3B is a perspective view of an end effector with a camera in analigned orientation, according to one or more embodiments of the presentdisclosure;

FIG. 4 is a side view of a ground-based visual-inspection apparatusinspecting an aircraft, according to one or more embodiments of thepresent disclosure;

FIG. 5 is a block diagram of a ground-based visual-inspection system,according to one or more embodiments of the present disclosure; and

FIG. 6 is a schematic flow diagram of a method, according to one or moreembodiments of the present disclosure.

DETAILED DESCRIPTION

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present disclosure.Appearances of the phrases “in one embodiment,” “in an embodiment,” andsimilar language throughout this specification may, but do notnecessarily, all refer to the same embodiment. Similarly, the use of theterm “implementation” means an implementation having a particularfeature, structure, or characteristic described in connection with oneor more embodiments of the present disclosure, however, absent anexpress correlation to indicate otherwise, an implementation may beassociated with one or more embodiments.

Referring to FIG. 1, one embodiment of a ground-based visual-inspectionapparatus 100 is shown. The ground-based visual-inspection apparatus 100is used to visually inspect structures, such as vehicles (e.g., anaircraft 122 as shown in FIGS. 2 and 4), by capturing high-resolutionimages of the structures and associating the captured images withrelevant information. As an example, aircraft are sometimes required tobe visually inspected for wear. The size and shape of many aircraft makeit difficult to visually inspect all necessary surface areas. Onesolution to the problem of visually inspecting difficult-to-reach areasis to utilize drones or unmanned aerial vehicles (UAVs) to fly close tothe aircraft and capture images of the aircraft.

Captured images from UAVs are sometimes blurry, which may occur becauseof vibration of the UAV or because of weather conditions that make itdifficult to maneuver the UAV. Landing the UAV on the aircraft may helpalleviate such concerns. Unfortunately, however, UAVs can land on onlysome areas of the aircraft. Moreover flying a UAV close enough to anaircraft to land on the aircraft increases the risk of causing damage tothe aircraft. For these and other reasons, a ground-basedvisual-inspection apparatus 100 and corresponding system are disclosed.

The ground-based visual-inspection apparatus 100 includes a mobile base102. The mobile base 102 is a cart capable of maneuvering and supportingthe remainder of the ground-based visual-inspection apparatus 100. Insome embodiments, the mobile base 102 may be maneuvered under manualpower. In other embodiments, the mobile base 102 may be maneuvered bymotorized power. The mobile base 102 includes wheels 103 that allowholonomic motion, i.e., the mobile base 102 may be moved in anytranslational direction while simultaneously turning, thus allowing themobile base 102 to be moved into any location near a structure to beinspected.

The ground-based visual-inspection apparatus 100 includes an actuatablearm 104. The actuatable arm 104 is a mechanical arm that is locatable inthree dimensions. In the illustrated embodiment, the actuatable arm 104includes a vertical actuator 108 and a horizontal actuator 109.Referring to FIG. 1 the vertical actuator 108 allows for movement in avertical direction as shown by arrows 132. That is, the verticalactuator 108 is configured to actuate to any height within apredetermined range of heights. The range of heights, in someembodiments, would be any height that is greater than the height of thestructure, allowing the actuatable arm 104 to position a camera 110 nearany surface of the structure. The vertical actuator 108 may be any typeof linear actuator, telescoping pole, a zippermast, an indexing rail, ahydraulic or pneumatic piston, electric motor, or other similaractuatable apparatus. In some embodiments, the system may be movedmanually using human powered mechanisms, such as cables. Although any ofvarious structures can be inspected by the apparatus 100, hereinafter,the features and functionality of the apparatus 100 will be described inassociation with the aircraft 122.

Similarly, the horizontal actuator 109 allows for movement in ahorizontal direction as shown by arrows 138. That is, the horizontalactuator 109 is configured to actuate to any length within apredetermined range of lengths. The range of lengths, in someembodiments, would be any length that is greater than the half the widthof the fuselage of the aircraft 122, allowing the actuatable arm 104 toposition a camera 110 near any surface of the aircraft 122. Thehorizontal actuator 109 may be any type of linear actuator, telescopingpole, a zippermast, an indexing rail, a hydraulic or pneumatic piston,electric motor, or other similar actuatable apparatus.

With the movement of the mobile base 102 and the vertical and horizontaldisplacement of the vertical actuator 108 and the horizontal actuator109, respectively, an end effector 106 coupled to the actuatable arm 104is locatable in any three dimensional location. In addition to themovement described above, it is contemplated that the actuatable arm 104may include further degrees of motion, such as, rotation of the verticalactuator 108 relative to the mobile base 102 (which is noted by arrows134), rotation of the horizontal actuator 109 relative to the verticalactuator 108 (which is noted by arrows 136, and rotation of the endeffector 106 relative to the horizontal actuator 109 (which is noted byarrows 139).

While the actuatable arm 104 is shown and described as a combination oflinear actuators, the actuatabale arm 104 may comprise otherconfigurations that allow for locating an end effector 106 in a threedimensional space, including but not limited to, a jib crane,telescoping crane, robotic arm, or a combination of linear and rotatingactuators etc.

Coupled to the end of the actuatable arm 104 is the end effector 106.The end effector 106 is rotatable relative to the horizontal actuator109 as shown by arrows 139. The rotation of the end effector 106 allowsfor the end effector 106 to be locatable (e.g., a positionable andorientatable) in a three dimensional space.

The end effector 106 includes a camera 110. The camera 110 may beconfigured to capture high-resolution video or still photography. As theend effector 106 is locatable in a three dimensional space, the positionand orientation of the camera 110 is known. Such information can beassociated with the captured images of the camera 110 allowing a personto know the location of the camera relative to the aircraft 122 and theorientation of the camera 110 relative to the aircraft 122.

As camera 110 is coupled to the mobile base 102 on the ground the camera110 may capture high-resolution images with less vibration than camerasassociated with UAVs. In addition, the camera 110 may be positionedclose to the surface of the aircraft 122 without inadvertently bumpingthe surface of the aircraft 122, which also may occur with camerasassociated with UAVs.

The ground-based visual-inspection apparatus 100 may further beassociated with a control system 150. The control system 150 may beconfigured to allow for a user to control the movement of the mobilebase 102, the actuatable arm 104, the end effector 106, and the camera110. In some embodiments, the control system 150 is operated via acomputer terminal 101. The computer terminal 101 may be configured toprovide various measurements to the user including, but not limited tolocation of the mobile base 102, orientation of the mobile base 102,height of the vertical actuator 108, length of the horizontal actuator109, orientation of the end effector 106. In other embodiments, thecontrol system 150 remotely controls the ground-based visual-inspectionapparatus 100.

The control system 150 is configured to associate location information128 on a large structure, such as an aircraft 122, with images 126captured of the structure. As described below, particular reference willbe made to inspection of an aircraft 122. However, it is recognized thatthe apparatus 100 and control system 150 can be used to inspect any ofvarious other large structures, such as vehicles, ships, rockets, etc.In some embodiments, the control system 150 is further configured todetermine a location of the camera 110 relative to a reference location125 on the aircraft 122. The location of the camera 110 may bedetermined by measuring the movement of the ground-basedvisual-inspection apparatus 100, for example, by optically measuringmovement of the camera 110 relative to a reference location 125(through, for example, simultaneous localization and mapping (SLAM)), orthe other measurements described herein. The location information 128may include a translational displacement of a mobile base 102 from areference position and rotational displacement of the mobile base 102from a reference location 125. A reference location 125 may include areference position and/or a reference orientation of the ground-basedvisual-inspection apparatus 100.

Referring now to FIG. 2, the ground-based visual-inspection apparatus100 is shown inspecting an aircraft 122. The ground-basedvisual-inspection apparatus 100 is capable of moving relative to theaircraft 122. The positioning of the ground-based visual-inspectionapparatus 100 may be measured relative to a reference location 125 ofthe ground-based visual-inspection apparatus 100 (such as a startinglocation). For example, the ground-based visual-inspection apparatus 100may be positioned and oriented at the nose 123 of the aircraft 122. Asthe ground-based visual-inspection apparatus 100 is moved, themeasurements are relative to the reference location 125. That is, thedisplacement and orientation of the mobile base 102 is measured relativeto the reference location 125. In addition, the vertical displacement ofthe vertical actuator 108 is measured as well as the displacement of thehorizontal actuator 109 and the rotation of the horizontal actuator 109relative to the vertical actuator 108. Furthermore, the orientation ofthe end effector 106 relative to the reference location 125 is measured.These measurements allow for the location of the captured image(s) to beknown on the aircraft 122. With a close-up high-resolution image, theimage itself likely has no visual reference points to indicate where onthe aircraft 122 the image has been captured.

The reference location 125 may be in relation to the nose 123 of theaircraft 122. The reference location 125 may be in relation to any otheridentifiable location on the aircraft 122, such as a wing, window,engine, stabilizer, flap, landing gear, or other identifiable componentor part of the aircraft 122.

The ground-based visual-inspection apparatus 100 may include varioussensors or encoders that are configured to measure the movement of thecomponents of the ground-based visual-inspection apparatus 100. Forexample, the mobile base 102 may include sensors that sense thedisplacement and rotation of the mobile base 102. In an implementation,the displacement and rotation of the mobile base 102 is measured by adifferential odometry system that is configured to measure the movementof the wheels 103 (e.g., by odometry). In another implementation,movement of the mobile base 102 is measured by a global positioningsystem (GPS). In another implementation, encoders and sensors are usedto measure movement of the mobile base 102. The ground-basedvisual-inspection apparatus 100 may use various components to measuremovement including an incremental encoder, absolute encoder, rotaryencoder, position sensor, proximity sensor, linear variable differentialtransformer, potentiometer, optical sensors, and other similar encodersor sensors.

The ground-based visual-inspection apparatus 100 may include varioussensors or encoders that are configured to measure the movement of theactuatable arm 104. The sensors may be configured to measure therelative movement of the individual components of the actuatable arm104, such as the vertical actuator 108 and the horizontal actuator 109.As an example, the vertical position of the vertical actuator 108 may bemeasured by an indexing rail or linear position sensors that sense thepositioning of the vertical actuator 108. The actuatable arm 104 may usevarious components to measure movement including an incremental encoder,absolute encoder, rotary encoder, position sensor, proximity sensor,linear variable differential transformer, potentiometer, opticalsensors, and other similar encoders or sensors.

Referring now to FIGS. 3A and 3B, an end effector 106 is shown. The endeffector 106 includes a camera 110 coupled to a support element 105 by apivot joint 107. The pivot joint 107 allows the camera 110 to moverelative to the support element 105. In some embodiments, the camera 110and the end effector 106 move relative to each other. That is, thecamera 110 rotates relative to other parts of the end effector 106. Insome embodiments, the camera 110 and the end effector 106 do not moverelative to each other. That is, the rotation of the end effector 106relative to the actuatable arm 104 rotates the camera 110.

The end effector 106 may further include an alignment system 170. Thealignment system 170 includes a plurality of laser rangefinders 140. Thelaser rangefinders 140 are coupled to the camera 110. The laserrangefinders 140 are configured to use lasers 141 to measure thepositioning of the camera 110 relative to the aircraft 122. In someembodiments, the laser rangefinders 140 are configured to measure thedistance of the camera 110 from the aircraft 122. In some embodiments,the laser rangefinders 140 are configured to measure the orientation ofthe camera 110 relative to the aircraft 122. The alignment system 170includes a plurality of laser rangefinders 140 which are positioned onopposite sides of the camera 110. The alignment system 170 may utilizetwo or more laser rangefinders 140. In some embodiment, the laserrangefinders 140 are disposed on opposite sides of the camera 110. Insome embodiments, the laser rangefinders 140 are spaced around aperimeter of the camera 110. The number and spacing of the laserrangefinders 140 may be configured to allow the alignment system 170 todetermine the orientation of the camera 110 relative to the aircraft122.

The alignment system 170 may utilize different types of rangefindersincluding, but not limited to, optical, laser, radar, sonar, lidar, andultrasonic proximity sensors.

In some embodiments, the alignment system 170 is configured to utilize afeedback system to automatically align the camera 110 to an orientationperpendicular to the surface of the aircraft 122 that is to be captured.In some embodiments, the alignment system 170 continuously adjusts thealignment of the camera 110 as the ground-based visual-inspectionapparatus 100 moves relative to the aircraft 122. As the ground-basedvisual-inspection apparatus 100 moves along the target area of theaircraft 122, the alignment system 170 may automatically align thecamera 110 by receiving data from the laser rangefinders 140 andcontrolling any actuators that may control movement of the ground-basedvisual-inspection apparatus 100 and its various components.

In some embodiments, the orientation of the camera 110 is manuallycontrolled. Referring to FIG. 3A, the camera 110 is misaligned with thesurface of the aircraft 122. The alignment system 170 rotates the camera110 to automatically adjust the orientation of the camera 110 until thelaser rangefinders 140 are lined up with the surface of the aircraft 122as depicted in FIG. 3B, with the lasers 141 all sensing the surface ofthe aircraft 122 a same or similar distance from the camera 110.

Referring now to FIG. 4, a ground-based visual-inspection system 50 andan aircraft 122 are shown. The ground-based visual-inspection system 50includes a ground-based visual-inspection apparatus 100 and an alignmentsystem 170. The alignment system 170 includes laser rangefinders 140which are utilizing lasers 141 to measure a distance to the aircraft122. The lasers 141 of the laser rangefinders 140 are utilized to orientthe camera 110 relative to the aircraft 122 by rotating the end effector106 until the lasers 141 indicate equal distance to the aircraft 122.

In some embodiments, the laser rangefinders 140 are configured tofunction as proximity sensors. As proximity sensors, the laserrangefinders 140 may assist in automated motion control of theground-based visual-inspection apparatus 100 to notify the ground-basedvisual-inspection apparatus 100 of proximity to the aircraft 122 and, insome embodiments, automatically activate/deactivate actuators. That is,data from the laser rangefinders 140 may be used to stop or activate thevarious actuators of the ground-based visual-inspection apparatus 100and automatically deter potential collisions. Other types of collisionpreventions systems may be deployed which are configured to preventcollisions between the ground-based visual-inspection apparatus 100 andthe aircraft 122. In some embodiments without automated control of theground-based visual-inspection apparatus 100, the laser rangefinders 140may activate indicator lights or a warning sound to warn a user of apotential collision between the ground-based visual-inspection apparatus100 and the aircraft 122.

As discussed previously, the ground-based visual-inspection apparatus100 may utilize a reference location 125 (starting location) and measuremovement of the ground-based visual-inspection apparatus 100 relative tothe reference location 125. In some embodiments, pre-acquired data ofthe aircraft 122 is utilized. In some embodiments, the pre-acquired datais retrieved from a computer aided design (CAD) model of the aircraft122.

Referring now to FIG. 5, a block diagram of a ground-basedvisual-inspection system 50 is shown. The ground-based visual-inspectionsystem 50 is configured to capture visual images to aid in visuallyinspection of aircraft. The ground-based visual-inspection system 50includes a ground-based visual-inspection apparatus 100 and a controlsystem 150.

The ground-based visual-inspection system 50 includes a ground-basedvisual-inspection apparatus 100 may include some or all of the featuresdescribed herein in conjunction with the remaining figures. The controlsystem 150 includes a locating system 160 and an alignment system 170.The control system 150 is configured to control and regulate themovement of the ground-based visual-inspection apparatus 100.

The control system 150 may include various components, not illustrated,to allow for control of the components of the ground-basedvisual-inspection apparatus 100 described herein, such as, but notlimited to, processors, memory, computer hardware and software,controllers, and modules. The control system 150 may be furtherconfigured to measure or receive inputs from sensors and encoders andadjust the ground-based visual-inspection apparatus 100 accordingly. Insome embodiments, the control system 150 receives data from sensors 127or laser rangefinders 140. The data from the sensors 127 or laserrangefinders 140 may be used as inputs to direct the ground-basedvisual-inspection apparatus 100 (and, more specifically, the individualactuators) to achieve alignment with the structure 122. In someembodiments, the data from sensors 127 or laser rangefinders 140 is usedto automatically adjust the angle and/or the distance of the camera 110relative to the surface of the structure 122.

The control system 150 includes, in some embodiments, a locating system160. The locating system 160 is configured to control the positioningand orientation of the ground-based visual-inspection apparatus 100. Insome embodiments, the locating system 160 is configured to continuallyadjust the position and orientation of the components of theground-based visual-inspection apparatus 100 while the ground-basedvisual-inspection apparatus 100 is moved over the target area of theaircraft 122. The locating system 160 may be configured to activateactuators (such as vertical actuator 108 and horizontal actuator 109,and the rotational actuator (not shown) that rotates the end effector106 about pivot joint 107) to adjust the position and orientation of theend effector 106 relative to the aircraft 122. As an example, as theground-based visual-inspection apparatus 100 moves along the fuselage ofthe aircraft 122, the locating system 160 may determine the distance tothe fuselage and automatically adjust the ground-based visual-inspectionapparatus 100 to keep the camera 110 at a consistent distance from thefuselage and orientation relative to the surface. In some embodiments,the locating system 160 is configured to measure the movement of theground-based visual-inspection apparatus 100. The locating system 160may be configured to measure the absolute movement of the ground-basedvisual-inspection apparatus 100 or the relative movement of theground-based visual-inspection apparatus 100 relative to a referencelocation 125.

In some embodiments, the locating system 160 is configured to controlthe positioning and orientation and measure the movement of the mobilebase 102 and determine a base location 162. The base location 162 may bean absolute location or a relative location relative to a startinglocation or reference location 125.

In some embodiments, the locating system 160 is configured to controlthe positioning and orientation and measure the movement of theactuatable arm 104 and determine an arm location 164. The arm location164 may be an absolute location or a location relative to a startinglocation or reference location 125.

In some embodiments, the locating system 160 is configured to controlthe positioning and orientation and measure the movement of the endeffector 106 and determine an end effector location 166. The endeffector location 166 may be an absolute location or a location relativeto a starting location or reference location 125.

The control system 150 includes, in some embodiments, an alignmentsystem 170. The alignment system 170 is configured to control thealignment of the end effector 106 and camera 110. In some embodiments,the alignment system 170 is configured to measure the movement andlocation of the end effector 106 and camera 110 relative to the aircraft122. The alignment system 170 can be configured in a manner similar toor the same as that of the alignment system described in more detail inU.S. patent application 15/623,304, filed Jun. 14, 2017.

In some embodiments, the alignment system 170 is configured to measureand record a distance information 172 of the camera 110 or end effector106 relative to the aircraft 122. Such distance information 172 may beassociated with captured images 126 of the aircraft 122. In someembodiments, the alignment system 170 is configured to measure andrecord orientation information 174 of the camera 110 or end effector 106relative to the aircraft 122. Such orientation information 174 may beassociated with the captured images 126 of the aircraft 122.

The distance information 172 and the orientation information 174 may beassociated or linked to the captured images 126 or may be embedded intothe captured images as location information 128. Location information128 may be metadata embedded into the captured images 126 may includeany information measured or noted by the control system 150 including,but not limited to, positioning information of the ground-basedvisual-inspection apparatus 100, location information of the capturedimage 126 on the aircraft (based on a reference location 125 or a CADmodel), the distance information 172, or the orientation information174. The information associated with the captured images 126 assist auser in evaluating the captured images 126 more accurately.

In some embodiments, the location information 128 is the currentlocation (i.e. position and orientation) of the camera 110 (or endeffector 106) defined in a local coordinate system of the structure 122.The local coordinate system of the structure 122 may be determined by aCAD model or other similar information of the structure 122. In someembodiments, an association between the captured images 126 and thecurrent location (i.e. position and orientation) of the camera 110 (orend effector 106) is determined. In some embodiments, the location ofthe camera 110, obtained from the location of the base 102 relative tothe reference location 125, relative to the reference location 125 inthe local coordinate system is combined with the kinematic movement ofthe actuatable arm 104 relative to the base 102.

In some embodiments, the control system 150 acquires a base location 162from either odometry and/or GPS. For example, a transformation matrix ofthe mobile base 102 relative to an origin or reference location 125 maybe acquired. In some embodiments, the location of the end effector 106or camera 110 is determined by encoder data and the geometry of theactuatable arm 104. As another example, a transformation matrix of theactuatable arm 104 relative to an origin or reference location 125 maybe computed to determine the arm location 164. In some embodiments,matrix multiplication of the two transformation matrices is performed tocompute the transformation matrix of the end effector 106 (and, byextension, the camera 110) relative to the origin of the localcoordinate system to compute the end effector location 166. In someembodiments, the control system 150 may then associate the locationinformation 128 of the camera 110 relative to the origin of the localcoordinate system of the target object with the current captured image126. In some embodiments, this could mean embedding the locationinformation 128 into the image 126 (for example, by using ExchangeableImage File Format (EXIF) for image metadata).

In some embodiments, the control system 150 includes sensors 127. Thesensors 127 may be part of the ground-based visual-inspection apparatus100. In some embodiments, some or all of the sensors 127 may beoff-board sensors. As an example, the sensors may be on the aircraft 122or on the ground, or both. The sensors 127 may be configured to providedata to the control system 150. The data may include distance andorientation information of the ground-based visual-inspection apparatus100 relative to the aircraft 122. The data may, in some embodiments, beused to control the movement of the ground-based visual-inspectionapparatus 100. That is, the motion of the ground-based visual-inspectionapparatus 100 may be based of the data of the sensors 127. Inembodiments that include off-board sensors, the control system 150 orground-based visual-inspection apparatus 100 may be configured tocommunicate with the off-board sensors. In some embodiments, the controlsystem 150 may be configured to wirelessly communicate with theoff-board sensors.

Now referring to FIG. 6, one embodiment of a method 600 is shown. Themethod 600 includes aligning 602 a ground-based visual-inspectionapparatus to an aircraft, the ground-based visual-inspection apparatuscomprising a mobile base, an actuatable arm coupled to the mobile base,and an end effector coupled to the actuatable arm, wherein the endeffector comprises a camera. The method 600 includes capturing 604 animage of the aircraft with the camera. The method 600 includesdetermining location information of the camera relative to a referencelocation by acquiring a base location, arm location, and end effectorlocation relative to a reference location. The method 600 includesassociating 608 location information with the image. The method thenends.

In some embodiments, the method 600 may further include embedding thelocation information into the image.

In certain embodiments, the method 600 may further include acquiring abase location by computing a transformation matrix of the mobile base102 relative to the reference location. In some embodiments, the method600 may further include acquiring an arm location by computing atransformation matrix of the actuatable arm relative to the referencelocation. In various embodiments, the method may further includeperforming matrix multiplication of the transformation matrix of themobile base and the transformation matrix of the actuatable arm tocompute a transformation matrix of the end effector relative to thereference location.

In some embodiments, the method may include determining distanceinformation of the camera relative to the aircraft and associating thedistance information with the image. In certain embodiments, the methodmay include determining orientation information of the camera relativeto the aircraft and associating the orientation information with theimage.

Although described in a depicted order, the method may proceed in any ofa number of ordered combinations.

In the above description, certain terms may be used such as “up,”“down,” “upper,” “lower,” “horizontal,” “vertical,” “left,” “right,”“over,” “under” and the like. These terms are used, where applicable, toprovide some clarity of description when dealing with relativerelationships. But, these terms are not intended to imply absoluterelationships, positions, and/or orientations. For example, with respectto an object, an “upper” surface can become a “lower” surface simply byturning the object over. Nevertheless, it is still the same object.Further, the terms “including,” “comprising,” “having,” and variationsthereof mean “including but not limited to” unless expressly specifiedotherwise. An enumerated listing of items does not imply that any or allof the items are mutually exclusive and/or mutually inclusive, unlessexpressly specified otherwise. The terms “a,” “an,” and “the” also referto “one or more” unless expressly specified otherwise. Further, the term“plurality” can be defined as “at least two.”

Additionally, instances in this specification where one element is“coupled” to another element can include direct and indirect coupling.Direct coupling can be defined as one element coupled to and in somecontact with another element. Indirect coupling can be defined ascoupling between two elements not in direct contact with each other, buthaving one or more additional elements between the coupled elements.Further, as used herein, securing one element to another element caninclude direct securing and indirect securing. Additionally, as usedherein, “adjacent” does not necessarily denote contact. For example, oneelement can be adjacent another element without being in contact withthat element.

As used herein, the phrase “at least one of”, when used with a list ofitems, means different combinations of one or more of the listed itemsmay be used and only one of the items in the list may be needed. Theitem may be a particular object, thing, or category. In other words, “atleast one of” means any combination of items or number of items may beused from the list, but not all of the items in the list may berequired. For example, “at least one of item A, item B, and item C” maymean item A; item A and item B; item B; item A, item B, and item C; oritem B and item C. In some cases, “at least one of item A, item B, anditem C” may mean, for example, without limitation, two of item A, one ofitem B, and ten of item C; four of item B and seven of item C; or someother suitable combination.

Unless otherwise indicated, the terms “first,” “second,” etc. are usedherein merely as labels, and are not intended to impose ordinal,positional, or hierarchical requirements on the items to which theseterms refer. Moreover, reference to, e.g., a “second” item does notrequire or preclude the existence of, e.g., a “first” or lower-numbereditem, and/or, e.g., a “third” or higher-numbered item.

As used herein, a system, apparatus, structure, article, element,component, or hardware “configured to” perform a specified function isindeed capable of performing the specified function without anyalteration, rather than merely having potential to perform the specifiedfunction after further modification. In other words, the system,apparatus, structure, article, element, component, or hardware“configured to” perform a specified function is specifically selected,created, implemented, utilized, programmed, and/or designed for thepurpose of performing the specified function. As used herein,“configured to” denotes existing characteristics of a system, apparatus,structure, article, element, component, or hardware which enable thesystem, apparatus, structure, article, element, component, or hardwareto perform the specified function without further modification. Forpurposes of this disclosure, a system, apparatus, structure, article,element, component, or hardware described as being “configured to”perform a particular function may additionally or alternatively bedescribed as being “adapted to” and/or as being “operative to” performthat function.

The schematic flow chart diagram included herein is generally set forthas logical flow chart diagrams. As such, the depicted order and labeledsteps are indicative of one embodiment of the presented method. Othersteps and methods may be conceived that are equivalent in function,logic, or effect to one or more steps, or portions thereof, of theillustrated method. Additionally, the format and symbols employed areprovided to explain the logical steps of the method and are understoodnot to limit the scope of the method. Although various arrow types andline types may be employed in the flow chart diagrams, they areunderstood not to limit the scope of the corresponding method. Indeed,some arrows or other connectors may be used to indicate only the logicalflow of the method. For instance, an arrow may indicate a waiting ormonitoring period of unspecified duration between enumerated steps ofthe depicted method. Additionally, the order in which a particularmethod occurs may or may not strictly adhere to the order of thecorresponding steps shown.

The control system, which may include associated modules and/orelectronic controllers, described in this specification may beimplemented as a hardware circuit comprising custom VLSI circuits orgate arrays, off-the-shelf semiconductors such as logic chips,transistors, or other discrete components. The control system may alsobe implemented in programmable hardware devices such as fieldprogrammable gate arrays, programmable array logic, programmable logicdevices or the like.

The control system may also be implemented in code and/or software forexecution by various types of processors. An identified module of codemay, for instance, comprise one or more physical or logical blocks ofexecutable code which may, for instance, be organized as an object,procedure, or function. Nevertheless, the executables of the electroniccontroller need not be physically located together, but may comprisedisparate instructions stored in different locations which, when joinedlogically together, comprise the electronic controller and achieve thestated purpose for the electronic controller.

Indeed, code of the electronic controller may be a single instruction,or many instructions, and may even be distributed over several differentcode segments, among different programs, and across several memorydevices. Similarly, operational data may be identified and illustratedherein within the electronic controller, and may be embodied in anysuitable form and organized within any suitable type of data structure.The operational data may be collected as a single data set, or may bedistributed over different locations including over different computerreadable storage devices. Where the electronic controller or portions ofthe electronic controller are implemented in software, the softwareportions are stored on one or more computer readable storage devices.

Any combination of one or more computer readable medium may be utilized.The computer readable medium may be a computer readable storage medium.The computer readable storage medium may be a storage device storing thecode. The storage device may be, for example, but not limited to, anelectronic, magnetic, optical, electromagnetic, infrared, holographic,micromechanical, or semiconductor system, apparatus, or device, or anysuitable combination of the foregoing.

More specific examples (a non-exhaustive list) of the storage devicewould include the following: an electrical connection having one or morewires, a portable computer diskette, a hard disk, a random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

Code for carrying out operations for embodiments may be written in anycombination of one or more programming languages including an objectoriented programming language such as Python, Ruby, Java, Smalltalk,C++, or the like, and conventional procedural programming languages,such as the “C” programming language, or the like, and/or machinelanguages such as assembly languages. The code may execute entirely onthe user's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

The described features, structures, or characteristics of theembodiments may be combined in any suitable manner. In the abovedescription, numerous specific details are provided, such as examples ofprogramming, software modules, user selections, network transactions,database queries, database structures, hardware modules, hardwarecircuits, hardware chips, etc., to provide a thorough understanding ofembodiments. One skilled in the relevant art will recognize, however,that embodiments may be practiced without one or more of the specificdetails, or with other methods, components, materials, and so forth. Inother instances, well-known structures, materials, or operations are notshown or described in detail to avoid obscuring aspects of anembodiment.

Aspects of the embodiments are described below with reference toschematic flowchart diagrams and/or schematic block diagrams of methods,apparatuses, systems, and program products according to embodiments. Itwill be understood that each block of the schematic flowchart diagramsand/or schematic block diagrams, and combinations of blocks in theschematic flowchart diagrams and/or schematic block diagrams, can beimplemented by code. These code may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the functions/acts specified in the schematic flowchartdiagrams and/or schematic block diagrams block or blocks.

The code may also be stored in a storage device that can direct acomputer, other programmable data processing apparatus, or other devicesto function in a particular manner, such that the instructions stored inthe storage device produce an article of manufacture includinginstructions which implement the function/act specified in the schematicflowchart diagrams and/or schematic block diagrams block or blocks.

The code may also be loaded onto a computer, other programmable dataprocessing apparatus, or other devices to cause a series of operationalsteps to be performed on the computer, other programmable apparatus orother devices to produce a computer implemented process such that thecode which execute on the computer or other programmable apparatusprovide processes for implementing the functions/acts specified in theflowchart and/or block diagram block or blocks.

The schematic flowchart diagrams and/or schematic block diagrams in theFigures illustrate the architecture, functionality, and operation ofpossible implementations of apparatuses, systems, methods and programproducts according to various embodiments. In this regard, each block inthe schematic flowchart diagrams and/or schematic block diagrams mayrepresent a module, segment, or portion of code, which comprises one ormore executable instructions of the code for implementing the specifiedlogical function(s).

It should also be noted that, in some alternative implementations, thefunctions noted in the block may occur out of the order noted in theFigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. Other steps and methods may be conceived that are equivalentin function, logic, or effect to one or more blocks, or portionsthereof, of the illustrated Figures.

The present subject matter may be embodied in other specific formswithout departing from its spirit or essential characteristics. Thedescribed embodiments are to be considered in all respects only asillustrative and not restrictive. All changes which come within themeaning and range of equivalency of the claims are to be embraced withintheir scope.

What is claimed is:
 1. A ground-based visual-inspection system forinspecting a structure, the ground-based visual-inspection systemcomprising: a ground-based visual-inspection apparatus comprising: amobile base; an actuatable arm coupled to the mobile base, theactuatable arm locatable in a three dimensional space; and an endeffector coupled to the actuatable arm, wherein the end effectorcomprises a camera configured to capture images of the structure; and acontrol system, wherein the control system is configured to determinelocation information of the camera relative to a reference location andassociate the location information with the images.
 2. The ground-basedvisual-inspection system according to claim 1, wherein the controlsystem is further configured to determine the location information bydetermining a location of the camera relative to the reference locationby: acquiring a base location by computing a transformation matrix ofthe mobile base relative to the reference location; acquiring an armlocation by computing a transformation matrix of the actuatable armrelative to the reference location; and performing matrix multiplicationof the transformation matrix of the mobile base and the transformationmatrix of the actuatable arm to compute a transformation matrix of theend effector relative to the reference location.
 3. The ground-basedvisual-inspection system according to claim 1, wherein the controlsystem further comprises an alignment system comprising a plurality oflaser rangefinders coupled to the end effector.
 4. The ground-basedvisual-inspection system according to claim 3, wherein the alignmentsystem is configured to determine an orientation of the camera relativeto the structure based on data of the plurality of laser rangefinders.5. The ground-based visual-inspection system according to claim 4,wherein the control system is further configured to embed orientationinformation of the camera in the images.
 6. The ground-basedvisual-inspection system according to claim 3, wherein the alignmentsystem is configured to determine a distance of the camera relative tothe aircraft based on data captured by the plurality of laserrangefinders.
 7. The ground-based visual-inspection system according toclaim 6, wherein the control system is further configured to embeddistance information in the images.
 8. The ground-basedvisual-inspection system according to claim 1, wherein the controlsystem is configured to embed the location information into the image.9. The ground-based visual-inspection system according to claim 8,wherein the location information is based on data from sensors on theground-based visual-inspection apparatus or the structure.
 10. Theground-based visual-inspection system according to claim 1, wherein thelocation information is based on a reference location on the structure.11. The ground-based visual-inspection system according to claim 1,wherein the mobile base comprises a cart maneuverable by manual power.12. The ground-based visual-inspection system according to claim 1,further comprising an alignment system comprising a plurality of laserrangefinders coupled to the end effector, wherein: the alignment systemis configured to determine an orientation and a distance of the camerarelative to the structure based on data of the plurality of laserrangefinders; the control system is further configured to embedorientation information and distance information with the images; andthe control system is further configured to embed the locationinformation into the image.
 13. The ground-based visual-inspectionsystem according to claim 12, wherein the location information comprisesa translational displacement from a reference position and rotationaldisplacement from a reference orientation of the mobile base relative tothe reference location.
 14. The ground-based visual-inspection systemaccording to claim 1, wherein the actuatable arm comprises an indexingrail.
 15. A ground-based visual-inspection apparatus comprising: amobile base; an actuatable arm coupled to the mobile base, theactuatable arm locatable in a three dimensional space; an end effectorcoupled to the actuatable arm, wherein the end effector comprises acamera configured to capture images of a structure; and a controlsystem, wherein the control system is configured to determine locationinformation of the camera relative to a reference location.
 16. Theapparatus according to claim 15, further comprising a plurality of laserrangefinders coupled to the end effector.
 17. A method comprising:aligning a ground-based visual-inspection apparatus to a structure, theground-based visual-inspection apparatus comprising a mobile base, anactuatable arm coupled to the mobile base, and an end effector coupledto the actuatable arm, wherein the end effector comprises a camera;capturing an image of the structure with the camera; determininglocation information of the camera relative to a reference location byacquiring a base location, arm location, and end effector locationrelative to a reference location; and associating the locationinformation with the image.
 18. The method according to claim 17,further comprising receiving data from sensors and automaticallyadjusting the orientation or distance of the camera relative to thestructure based on the received data.
 19. The method according to claim17, further comprising: acquiring the base location by computing atransformation matrix of the mobile base relative to the referencelocation; acquiring the arm location by computing a transformationmatrix of the actuatable arm relative to the reference location; andperforming matrix multiplication of the transformation matrix of themobile base and the transformation matrix of the actuatable arm tocompute a transformation matrix of the end effector relative to thereference location.
 20. The method according to claim 17, furthercomprising: determining distance information and orientation informationof the camera relative to the structure; and associating the distanceinformation and the orientation information with the image.